{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visual Comparison Between Different Classification Methods in Shogun"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notebook by Youssef Emad El-Din (Github ID: <a href=\"https://github.com/youssef-emad/\">youssef-emad</a>)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrates different classification methods in Shogun. The point is to compare and visualize the decision boundaries of different classifiers on two different datasets, where one is linear seperable, and one is not.\n",
    "\n",
    "1. <a href =\"#section1\">Data Generation and Visualization</a>\n",
    "2. <a href =\"#section2\">Support Vector Machine</a>\n",
    " 1. <a href =\"#section2a\">Linear SVM</a>\n",
    " 2. <a href =\"#section2b\">Gaussian Kernel</a>\n",
    " 3. <a href =\"#section2c\">Sigmoid Kernel</a>\n",
    " 4. <a href =\"#section2d\">Polynomial Kernel</a>\n",
    "3. <a href =\"#section3\">Naive Bayes</a>\n",
    "4. <a href =\"#section4\">Nearest Neighbors</a>\n",
    "5. <a href =\"#section5\">Linear Discriminant Analysis</a>\n",
    "6. <a href =\"#section6\">Quadratic Discriminat Analysis</a>\n",
    "7. <a href =\"#section7\">Gaussian Process</a>\n",
    " 1. <a href =\"#section7a\">Logit Likelihood model</a>\n",
    " 2. <a href =\"#section7b\">Probit Likelihood model</a>\n",
    "8. <a href =\"#section8\">Putting It All Together</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from modshogun import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Needed lists for the final plot\n",
    "classifiers_linear = []*10\n",
    "classifiers_non_linear = []*10\n",
    "classifiers_names = []*10\n",
    "fadings = []*10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id = \"section1\">Data Generation and Visualization</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transformation of features to Shogun format using <a href=\"http://www.shogun-toolbox.org/doc/en/current/classshogun_1_1CDenseFeatures.html\">RealFeatures</a> and <a href=\"http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CBinaryLabels.html\">BinaryLables</a> classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "shogun_feats_linear = RealFeatures(CSVFile('../../../data/toy/classifier_binary_2d_linear_features_train.dat'))\n",
    "shogun_labels_linear = BinaryLabels(CSVFile('../../../data/toy/classifier_binary_2d_linear_labels_train.dat'))\n",
    "\n",
    "shogun_feats_non_linear = RealFeatures(CSVFile('../../../data/toy/classifier_binary_2d_nonlinear_features_train.dat'))\n",
    "shogun_labels_non_linear = BinaryLabels(CSVFile('../../../data/toy/classifier_binary_2d_nonlinear_labels_train.dat'))\n",
    "\n",
    "feats_linear = shogun_feats_linear.get_feature_matrix()\n",
    "labels_linear = shogun_labels_linear.get_labels()\n",
    "\n",
    "feats_non_linear = shogun_feats_non_linear.get_feature_matrix()\n",
    "labels_non_linear = shogun_labels_non_linear.get_labels()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data visualization methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_binary_data(plot,X_train, y_train):\n",
    "    \"\"\"\n",
    "    This function plots 2D binary data with different colors for different labels.\n",
    "    \"\"\"\n",
    "    plot.xlabel(r\"$x$\")\n",
    "    plot.ylabel(r\"$y$\")\n",
    "    plot.plot(X_train[0, np.argwhere(y_train == 1)], X_train[1, np.argwhere(y_train == 1)], 'ro')\n",
    "    plot.plot(X_train[0, np.argwhere(y_train == -1)], X_train[1, np.argwhere(y_train == -1)], 'bo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_plot_isolines(classifier,features,size=200,fading=True):\n",
    "    \"\"\"\n",
    "    This function computes the classification of points on the grid\n",
    "    to get the decision boundaries used in plotting\n",
    "    \"\"\"\n",
    "    x1 = np.linspace(1.2*min(features[0]), 1.2*max(features[0]), size)\n",
    "    x2 = np.linspace(1.2*min(features[1]), 1.2*max(features[1]), size)\n",
    "\n",
    "    x, y = np.meshgrid(x1, x2)\n",
    "\n",
    "    plot_features=RealFeatures(np.array((np.ravel(x), np.ravel(y))))\n",
    "    \n",
    "    if fading == True:\n",
    "        plot_labels = classifier.apply(plot_features).get_values()\n",
    "    else:\n",
    "        plot_labels = classifier.apply(plot_features).get_labels()\n",
    "    z = plot_labels.reshape((size, size))\n",
    "    return x,y,z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def plot_model(plot,classifier,features,labels,fading=True):\n",
    "    \"\"\"\n",
    "    This function plots an input classification model\n",
    "    \"\"\"\n",
    "    x,y,z = compute_plot_isolines(classifier,features,fading=fading)\n",
    "    plot.pcolor(x,y,z,cmap='RdBu_r')\n",
    "    plot.contour(x, y, z, linewidths=1, colors='black')\n",
    "    plot_binary_data(plot,features, labels)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.title(\"Linear Features\")\n",
    "plot_binary_data(plt,feats_linear, labels_linear)\n",
    "plt.subplot(122)\n",
    "plt.title(\"Non Linear Features\")\n",
    "plot_binary_data(plt,feats_non_linear, labels_non_linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id=\"section2\" href=\"http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CSVM.html\">Support Vector Machine</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section2a\" href=\"http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CLibLinear.html\">Linear SVM</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shogun provide <a href=\"http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CLibLinear.html\">Liblinear</a> which is a library for large-scale linear learning focusing on SVM used for classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,5))\n",
    "c  = 0.5\n",
    "epsilon =1e-3\n",
    "\n",
    "svm_linear = LibLinear(c,shogun_feats_linear,shogun_labels_linear)\n",
    "svm_linear.set_liblinear_solver_type(L2R_L2LOSS_SVC) \n",
    "svm_linear.set_epsilon(epsilon)\n",
    "svm_linear.train()\n",
    "classifiers_linear.append(svm_linear)\n",
    "classifiers_names.append(\"SVM Linear\")\n",
    "fadings.append(True)\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.title(\"Linear SVM - Linear Features\")\n",
    "plot_model(plt,svm_linear,feats_linear,labels_linear)\n",
    "\n",
    "svm_non_linear = LibLinear(c,shogun_feats_non_linear,shogun_labels_non_linear)\n",
    "svm_non_linear.set_liblinear_solver_type(L2R_L2LOSS_SVC) \n",
    "svm_non_linear.set_epsilon(epsilon)\n",
    "svm_non_linear.train()\n",
    "classifiers_non_linear.append(svm_non_linear)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.title(\"Linear SVM - Non Linear Features\")\n",
    "plot_model(plt,svm_non_linear,feats_non_linear,labels_non_linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVM - Kernels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shogun provides many options for using kernel functions. Kernels in Shogun are based on two classes which are <a href=\"http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CKernel.html\">CKernel</a> and <a href=\"http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CKernelMachine.html\">CKernelMachine</a> base class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id =\"section2b\" href = \"http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CGaussianKernel.html\">Gaussian Kernel</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gaussian_c=0.7\n",
    "\n",
    "gaussian_kernel_linear=GaussianKernel(shogun_feats_linear, shogun_feats_linear, 100)\n",
    "gaussian_svm_linear=LibSVM(gaussian_c, gaussian_kernel_linear, shogun_labels_linear)\n",
    "gaussian_svm_linear.train()\n",
    "classifiers_linear.append(gaussian_svm_linear)\n",
    "fadings.append(True)\n",
    "\n",
    "gaussian_kernel_non_linear=GaussianKernel(shogun_feats_non_linear, shogun_feats_non_linear, 100)\n",
    "gaussian_svm_non_linear=LibSVM(gaussian_c, gaussian_kernel_non_linear, shogun_labels_non_linear)\n",
    "gaussian_svm_non_linear.train()\n",
    "classifiers_non_linear.append(gaussian_svm_non_linear)\n",
    "classifiers_names.append(\"SVM Gaussian Kernel\")\n",
    "\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.title(\"SVM Gaussian Kernel - Linear Features\")\n",
    "plot_model(plt,gaussian_svm_linear,feats_linear,labels_linear)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.title(\"SVM Gaussian Kernel - Non Linear Features\")\n",
    "plot_model(plt,gaussian_svm_non_linear,feats_non_linear,labels_non_linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<a id =\"section2c\" href=\"http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CSigmoidKernel.html\">Sigmoid Kernel</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sigmoid_c = 0.9\n",
    "\n",
    "sigmoid_kernel_linear = SigmoidKernel(shogun_feats_linear,shogun_feats_linear,200,1,0.5)\n",
    "sigmoid_svm_linear = LibSVM(sigmoid_c, sigmoid_kernel_linear, shogun_labels_linear)\n",
    "sigmoid_svm_linear.train()\n",
    "classifiers_linear.append(sigmoid_svm_linear)\n",
    "classifiers_names.append(\"SVM Sigmoid Kernel\")\n",
    "fadings.append(True)\n",
    "\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.title(\"SVM Sigmoid Kernel - Linear Features\")\n",
    "plot_model(plt,sigmoid_svm_linear,feats_linear,labels_linear)\n",
    "\n",
    "sigmoid_kernel_non_linear = SigmoidKernel(shogun_feats_non_linear,shogun_feats_non_linear,400,2.5,2)\n",
    "sigmoid_svm_non_linear = LibSVM(sigmoid_c, sigmoid_kernel_non_linear, shogun_labels_non_linear)\n",
    "sigmoid_svm_non_linear.train()\n",
    "classifiers_non_linear.append(sigmoid_svm_non_linear)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.title(\"SVM Sigmoid Kernel - Non Linear Features\")\n",
    "plot_model(plt,sigmoid_svm_non_linear,feats_non_linear,labels_non_linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id =\"section2d\" href=\"http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CPolyKernel.html\">Polynomial Kernel</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "poly_c = 0.5\n",
    "degree = 4\n",
    "\n",
    "poly_kernel_linear = PolyKernel(shogun_feats_linear, shogun_feats_linear, degree, True)\n",
    "poly_svm_linear = LibSVM(poly_c, poly_kernel_linear, shogun_labels_linear)\n",
    "poly_svm_linear.train()\n",
    "classifiers_linear.append(poly_svm_linear)\n",
    "classifiers_names.append(\"SVM Polynomial kernel\")\n",
    "fadings.append(True)\n",
    "\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.title(\"SVM Polynomial Kernel - Linear Features\")\n",
    "plot_model(plt,poly_svm_linear,feats_linear,labels_linear)\n",
    "\n",
    "poly_kernel_non_linear=PolyKernel(shogun_feats_non_linear, shogun_feats_non_linear, degree, True)\n",
    "poly_svm_non_linear = LibSVM(poly_c, poly_kernel_non_linear, shogun_labels_non_linear)\n",
    "poly_svm_non_linear.train()\n",
    "classifiers_non_linear.append(poly_svm_non_linear)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.title(\"SVM Polynomial Kernel - Non Linear Features\")\n",
    "plot_model(plt,poly_svm_non_linear,feats_non_linear,labels_non_linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id =\"section3\" href=\"http://www.shogun-toolbox.org/doc/en/latest/classshogun_1_1CGaussianNaiveBayes.html\">Naive Bayes</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "multiclass_labels_linear = shogun_labels_linear.get_labels()\n",
    "for i in range(0,len(multiclass_labels_linear)):\n",
    "    if multiclass_labels_linear[i] == -1:\n",
    "        multiclass_labels_linear[i] = 0\n",
    "\n",
    "multiclass_labels_non_linear = shogun_labels_non_linear.get_labels()\n",
    "for i in range(0,len(multiclass_labels_non_linear)):\n",
    "    if multiclass_labels_non_linear[i] == -1:\n",
    "        multiclass_labels_non_linear[i] = 0\n",
    "\n",
    "\n",
    "shogun_multiclass_labels_linear = MulticlassLabels(multiclass_labels_linear)\n",
    "shogun_multiclass_labels_non_linear = MulticlassLabels(multiclass_labels_non_linear)\n",
    "\n",
    "naive_bayes_linear = GaussianNaiveBayes()\n",
    "naive_bayes_linear.set_features(shogun_feats_linear)\n",
    "naive_bayes_linear.set_labels(shogun_multiclass_labels_linear)\n",
    "naive_bayes_linear.train()\n",
    "classifiers_linear.append(naive_bayes_linear)\n",
    "classifiers_names.append(\"Naive Bayes\")\n",
    "fadings.append(False)\n",
    "\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.title(\"Naive Bayes - Linear Features\")\n",
    "plot_model(plt,naive_bayes_linear,feats_linear,labels_linear,fading=False)\n",
    "\n",
    "naive_bayes_non_linear = GaussianNaiveBayes()\n",
    "naive_bayes_non_linear.set_features(shogun_feats_non_linear)\n",
    "naive_bayes_non_linear.set_labels(shogun_multiclass_labels_non_linear)\n",
    "naive_bayes_non_linear.train()\n",
    "classifiers_non_linear.append(naive_bayes_non_linear)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.title(\"Naive Bayes - Non Linear Features\")\n",
    "plot_model(plt,naive_bayes_non_linear,feats_non_linear,labels_non_linear,fading=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id =\"section4\" href=\"http://www.shogun-toolbox.org/doc/en/current/classshogun_1_1CKNN.html\">Nearest Neighbors</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "number_of_neighbors = 10\n",
    "\n",
    "distances_linear = EuclideanDistance(shogun_feats_linear, shogun_feats_linear)\n",
    "knn_linear = KNN(number_of_neighbors,distances_linear,shogun_labels_linear)\n",
    "knn_linear.train()\n",
    "classifiers_linear.append(knn_linear)\n",
    "classifiers_names.append(\"Nearest Neighbors\")\n",
    "fadings.append(False)\n",
    "\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.title(\"Nearest Neighbors - Linear Features\")\n",
    "plot_model(plt,knn_linear,feats_linear,labels_linear,fading=False)\n",
    "\n",
    "distances_non_linear = EuclideanDistance(shogun_feats_non_linear, shogun_feats_non_linear)\n",
    "knn_non_linear = KNN(number_of_neighbors,distances_non_linear,shogun_labels_non_linear)\n",
    "knn_non_linear.train()\n",
    "classifiers_non_linear.append(knn_non_linear)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.title(\"Nearest Neighbors - Non Linear Features\")\n",
    "plot_model(plt,knn_non_linear,feats_non_linear,labels_non_linear,fading=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id =\"section5\" href=\"http://www.shogun-toolbox.org/doc/en/current/classshogun_1_1CLDA.html\">Linear Discriminant Analysis</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gamma = 0.1\n",
    "\n",
    "lda_linear = LDA(gamma, shogun_feats_linear, shogun_labels_linear)\n",
    "lda_linear.train()\n",
    "classifiers_linear.append(lda_linear)\n",
    "classifiers_names.append(\"LDA\")\n",
    "fadings.append(True)\n",
    "\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.title(\"LDA - Linear Features\")\n",
    "plot_model(plt,lda_linear,feats_linear,labels_linear)\n",
    "\n",
    "lda_non_linear = LDA(gamma, shogun_feats_non_linear, shogun_labels_non_linear)\n",
    "lda_non_linear.train()\n",
    "classifiers_non_linear.append(lda_non_linear)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.title(\"LDA - Non Linear Features\")\n",
    "plot_model(plt,lda_non_linear,feats_non_linear,labels_non_linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id =\"section6\" href=\"http://www.shogun-toolbox.org/doc/en/current/classshogun_1_1CQDA.html\">Quadratic Discriminant Analysis</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "qda_linear = QDA(shogun_feats_linear, shogun_multiclass_labels_linear)\n",
    "qda_linear.train()\n",
    "classifiers_linear.append(qda_linear)\n",
    "classifiers_names.append(\"QDA\")\n",
    "fadings.append(False)\n",
    "\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.title(\"QDA - Linear Features\")\n",
    "plot_model(plt,qda_linear,feats_linear,labels_linear,fading=False)\n",
    "\n",
    "qda_non_linear = QDA(shogun_feats_non_linear, shogun_multiclass_labels_non_linear)\n",
    "qda_non_linear.train()\n",
    "classifiers_non_linear.append(qda_non_linear)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.title(\"QDA - Non Linear Features\")\n",
    "plot_model(plt,qda_non_linear,feats_non_linear,labels_non_linear,fading=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id =\"section7\" href=\"http://www.shogun-toolbox.org/doc/en/current/classshogun_1_1CGaussianProcessBinaryClassification.html\">Gaussian Process</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id =\"section7a\">Logit Likelihood model</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shogun's <a href= \"http://www.shogun-toolbox.org/doc/en/current/classshogun_1_1CLogitLikelihood.html\">CLogitLikelihood</a> and <a href=\"http://www.shogun-toolbox.org/doc/en/current/classshogun_1_1CEPInferenceMethod.html\">CEPInferenceMethod</a> classes are used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# create Gaussian kernel with width = 2.0\n",
    "kernel = GaussianKernel(10, 2)\n",
    "# create zero mean function\n",
    "zero_mean = ZeroMean()\n",
    "# create logit likelihood model\n",
    "likelihood = LogitLikelihood()\n",
    "# specify EP approximation inference method\n",
    "inference_model_linear = EPInferenceMethod(kernel, shogun_feats_linear, zero_mean, shogun_labels_linear, likelihood)\n",
    "# create and train GP classifier, which uses Laplace approximation\n",
    "gaussian_logit_linear = GaussianProcessClassification(inference_model_linear)\n",
    "gaussian_logit_linear.train()\n",
    "classifiers_linear.append(gaussian_logit_linear)\n",
    "classifiers_names.append(\"Gaussian Process Logit\")\n",
    "fadings.append(True)\n",
    "\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.title(\"Gaussian Process - Logit - Linear Features\")\n",
    "plot_model(plt,gaussian_logit_linear,feats_linear,labels_linear)\n",
    "\n",
    "inference_model_non_linear = EPInferenceMethod(kernel, shogun_feats_non_linear, zero_mean,\n",
    "                                               shogun_labels_non_linear, likelihood)\n",
    "gaussian_logit_non_linear = GaussianProcessClassification(inference_model_non_linear)\n",
    "gaussian_logit_non_linear.train()\n",
    "classifiers_non_linear.append(gaussian_logit_non_linear)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.title(\"Gaussian Process - Logit - Non Linear Features\")\n",
    "plot_model(plt,gaussian_logit_non_linear,feats_non_linear,labels_non_linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id =\"section7b\">Probit Likelihood model</a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Shogun's <a href=\"http://www.shogun-toolbox.org/doc/en/current/classshogun_1_1CProbitLikelihood.html\">CProbitLikelihood</a> class is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "likelihood = ProbitLikelihood()\n",
    "\n",
    "inference_model_linear = EPInferenceMethod(kernel, shogun_feats_linear, zero_mean, shogun_labels_linear, likelihood)\n",
    "gaussian_probit_linear = GaussianProcessClassification(inference_model_linear)\n",
    "gaussian_probit_linear.train()\n",
    "classifiers_linear.append(gaussian_probit_linear)\n",
    "classifiers_names.append(\"Gaussian Process Probit\")\n",
    "fadings.append(True)\n",
    "\n",
    "plt.figure(figsize=(15,5))\n",
    "plt.subplot(121)\n",
    "plt.title(\"Gaussian Process - Probit - Linear Features\")\n",
    "plot_model(plt,gaussian_probit_linear,feats_linear,labels_linear)\n",
    "\n",
    "inference_model_non_linear = EPInferenceMethod(kernel, shogun_feats_non_linear,\n",
    "                                               zero_mean, shogun_labels_non_linear, likelihood)\n",
    "gaussian_probit_non_linear = GaussianProcessClassification(inference_model_non_linear)\n",
    "gaussian_probit_non_linear.train()\n",
    "classifiers_non_linear.append(gaussian_probit_non_linear)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.title(\"Gaussian Process - Probit - Non Linear Features\")\n",
    "plot_model(plt,gaussian_probit_non_linear,feats_non_linear,labels_non_linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id=\"section8\">Putting It All Together</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "figure = plt.figure(figsize=(30,9))\n",
    "plt.subplot(2,11,1)\n",
    "plot_binary_data(plt,feats_linear, labels_linear)\n",
    "for i in range(0,10):\n",
    "    plt.subplot(2,11,i+2)\n",
    "    plt.title(classifiers_names[i])\n",
    "    plot_model(plt,classifiers_linear[i],feats_linear,labels_linear,fading=fadings[i])\n",
    "\n",
    "plt.subplot(2,11,12)\n",
    "plot_binary_data(plt,feats_non_linear, labels_non_linear)\n",
    "\n",
    "for i in range(0,10):\n",
    "    plt.subplot(2,11,13+i)\n",
    "    plot_model(plt,classifiers_non_linear[i],feats_non_linear,labels_non_linear,fading=fadings[i])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
